Metadata Capture for Screen Sharing

ABSTRACT

A method for metadata capture for screen sharing of a graphical user interface (GUI) screen includes determining by a screen share sending module a region of the GUI screen to be shared; analyzing the GUI screen by a metadata capture module to determine metadata relevant to the region to be shared; and sending a screen share of the region comprising the relevant metadata to a receiver by the screen share sending module.

BACKGROUND

This disclosure relates generally to the field of screen sharing duringonline communication.

Screen sharing may be used to communicate information from a user's (orsender) graphical user interface (GUI) screen in, for example, an onlinechat or web conference. The screen share may comprise an image of theuser's GUI screen that is sent to a receiver. The GUI screen maycomprise metadata, including but not limited to confidentialityinformation, audit trail information, text, table cell color, a uniformresource locator (URL) link, a search query, or a persistent session ID.The metadata is not captured in the screen share image; for example,metadata such as selectable text or clickable URL links may be lost. Themetadata must be captured and shared separately.

Different types of metadata require different capturing methods. Forexample, text may be highlighted and copied, table cell color may beextracted from HTML code, a search query may be copied from a searchtextbox, a URL link may be right clicked, and “Copy Link Location”selected, a screen capture of a bitmap may be extracted by pushing theprintscreen key, or a session ID may be invisible, with no availableextraction method. A sender of a screen share may not be able to extractmore than one of type of metadata at a time. The receiver may determinesome metadata via post processing and analysis of received screen shareimage, using, for example, optical character recognition (OCR) orlexicographic analysis. However, such analysis does not capture alltypes of metadata that are present in the sender's GUI screen at thetime of the screen share, and may perform inefficiently or inaccurately.Therefore, in an instant messaging scenario, the sender may need to copyand paste the metadata information separately and manually to thereceiving party, which limits the usefulness of the screen share. In aweb conference scenario, the meeting host may not be able to copy andpaste metadata without interrupting the realtime nature of the webconference.

SUMMARY

An exemplary embodiment of a method for metadata capture for screensharing of a graphical user interface (GUI) screen includes determiningby a screen share sending module a region of the GUI screen to beshared; analyzing the GUI screen by a metadata capture module todetermine metadata relevant to the region to be shared; and sending ascreen share of the region comprising the relevant metadata to areceiver by the screen share sending module.

An exemplary embodiment of a system for metadata capture for screensharing of a graphical user interface (GUI) screen includes a screenshare sending module configured to determine a region of the GUI screento be shared; a metadata capture module configured to analyze the GUIscreen to determine metadata relevant to the region to be shared; andwherein the screen share sending module is further configured to send ascreen share of the region comprising the relevant metadata to areceiver.

An exemplary embodiment of a computer program product comprising acomputer readable storage medium containing computer code that, whenexecuted by a computer, implements a method for metadata capture forscreen sharing of a graphical user interface (GUI) screen, wherein themethod includes determining by a screen share sending module a region ofthe GUI screen to be shared; analyzing the GUI screen by a metadatacapture module to determine metadata relevant to the region to beshared; and sending a screen share of the region comprising the relevantmetadata to a receiver by the screen share sending module.

Additional features are realized through the techniques of the presentexemplary embodiment. Other embodiments are described in detail hereinand are considered a part of what is claimed. For a better understandingof the features of the exemplary embodiment, refer to the descriptionand to the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Referring now to the drawings wherein like elements are numbered alikein the several FIGURES:

FIG. 1 illustrates an embodiment of a system for metadata capture forscreen sharing.

FIG. 2 illustrates an embodiment of a method for metadata capture forscreen sharing.

FIG. 3 illustrates an embodiment of a GUI screen with a region selectedfor screen sharing.

FIG. 4 illustrates an embodiment of a sender interface for metadatacapture for screen sharing.

FIG. 5 illustrates an embodiment of a receiver interface for metadatacapture for screen sharing.

FIG. 6 illustrates an embodiment of a computer that may be used inconjunction with systems and methods for metadata capture for screensharing.

DETAILED DESCRIPTION

Embodiments of systems and methods for metadata capture for screensharing are provided, with exemplary embodiments being discussed belowin detail.

Capture of metadata may be performed automatically in a screen share ofa GUI screen, rather than through manual copying and pasting by thesender. The sender may select a region within the GUI screen for whichmetadata is to be extracted, or the sender may select the entire GUIscreen. Once a region of the screen is selected, object recognitiontechnology is applied to the selected region, and to objects outside ofthe selected region that are relevant to the selected region, torecognize and capture the metadata. Some or all of the captured metadatamay be sent to a receiver along with the screen share, according to thesender's preferences. The metadata may be rendered in the receiver'ssystem as HTML, an image map, native object, or any other desiredformat. The captured metadata is ready for use by the receiver, withoutthe need for additional processing. Metadata may be captured acrosstraditional boundaries, such as windows, frames, images or tables, andfrom outside of the designated metadata capture region. The metadata maybe captured and shared in a real-time, collaborative environment, toenhance and simplify collaboration and information sharing.

Metadata may include but is not limited to authorship information,confidentiality information, audit trail information, table cell color,a persistent session ID, text (including characteristics such ascontent, font size, or line spacing), window information such as titleor whether the window is minimized/maximized, or any other applicationspecific information, such as a search string URL for a browserapplication. Shared metadata may be re-rendered in the most appropriateformat with respect to the application from which the metadata isextracted. For example, captured metadata may be presented to appear asif it is on its native application. Application specific support forapplication windows in the screen share may be provided throughapplication specific plugins. For example, when sharing a Power Point®presentation or Microsoft Office® document, metadata such as author orlast modified date may be shared. When sharing a browser window, amechanism may be used that that introspects the HTML document describedby a standard format such as document object model (DOM). When sharing awebpage with persistence and shareable sessions, the URL to access thesession may also be sent as metadata.

FIG. 1 illustrates an embodiment of a system 100 for capturing metadatafor screen sharing. Sending computer system 101 comprises a GUI screen102. A user may select some or all of GUI screen 102 for a screen share.Screen share sending module 103 invokes metadata capture module 104 toidentify the metadata present in the selected region of GUI screen 102.Screen share sending module 103 then applies the user's preferences tothe identified metadata, and shares the screen share with the desiredmetadata with receiving computer system 106 via network 105. The screenshare and metadata are displayed to the receiver via screen sharereceiving module 107, which may comprise a chat or web conferencingapplication in some embodiments.

Metadata capture module 104 may implement any appropriate GUI objectnavigation technology to determine properties such as location, size,content, and visibility of objects in the GUI screen 102. By determiningeach object's coordinates in the GUI screen, metadata capture module 104may determine if an object is within the selected region, partiallywithin the region, or relevant to the selected region. The metadatacapture module 104 may then capture metadata from objects determined tobe within or relevant to the selected region. Metadata capture module104 may implement technology including but not limited to object mapfrom the .NET framework, Rational Function Tester, Spy++, PublicApplication Programming Interface (API), Microsoft Foundation Class(MFC), or SpyLib in some embodiments.

FIG. 2 illustrates an embodiment of a method 200 for capturing metadatafor screen sharing. FIG. 2 is discussed with reference to FIG. 1. Inblock 201, a sender selects a region of a GUI screen 102 to be shared asa screen share. The selected region may comprise a portion of GUI screen102, or all of GUI screen 102. In block 202, the selected region isanalyzed by metadata capture module 104 to identify any metadata presentin the selected region, and in any region of the GUI screen 102 relevantto the selected region if the selected region is less than the entireGUI screen 102. Data not currently visible in the GUI screen 102, butthat may be made visible by scrolling, may also be captured. In block203, the sender's preferences regarding which metadata to share in thescreen share are determined and applied to the identified metadata todetermine the metadata to be shared by screen share sending module 103.The sender preferences may be entered by the user at the time of thescreen share, by, for example, displaying a list of available metadatato the user, or the preferences may comprise a previously stored set ofpreferences. In block 204, the screen share is transmitted by screenshare sending module 103 to screen share receiving module 107 onreceiving computer system 106 along with the metadata to be shared thatwas determined in block 203.

Metadata may be captured from any portion of the GUI screen 102 that isdetermined to be relevant to the region of the GUI screen that isselected in block 201. FIG. 3 shows an embodiment of a GUI screen 300with a region 305 of the GUI screen 300 selected for metadata capturing.GUI screen 300 comprises various objects, including icons 301 a-d andwindows 302-304. The selected region 305 may be selected in anyappropriate manner, for example, clicking and dragging a mouse to form abox on a computer screen, or by making a circular region with finger ona touch-screen device in some embodiments. A selected region 305 maycomprise any portion or all of GUI screen 300. In FIG. 3, selectedregion 305 includes portions of windows 302 and 304. Therefore, in ametadata capture for a screen share of region 305, any metadataregarding windows 302 and 304 may be captured, whether the metadata isactually located inside of region 305 or not, as any metadata regardingwindows 302 and 304 is relevant to objects located inside region 305.Information regarding icons 301 a-d and window 303, however, may not becaptured.

The sender may designate preferences regarding the type and amount ofmetadata from selected region 305 to share. This may be accomplished viaa sender interface such as is shown in FIG. 4. FIG. 4 illustrates anembodiment of a sender interface 400 for capturing metadata. Interface400 may be launched when a sender selects a region of a GUI screen formetadata capture; however, interface 400 is shown for illustrativepurposes only, metadata capture may be performed via any appropriateinterface. Interface 400 comprises MyCapture 401, which corresponds tothe selected region of the sender's GUI screen, such as selected region305 of FIG. 3. MyCapture 401 comprises a web browser application havinga URL 402, a search string 403, and various clickable links such asclickable link 404. MyCapture 401 also comprises applicationsMyDisclosures 405 and Infoprint Footprint Manager 406. Availablemetadata window 407 lists the metadata available in MyCapture 401, andallows the sender to use checkboxes to designate which of the listedmetadata to send in a screen share of MyCapture 401. Once the user hasselected the desired metadata for sending with the screen share, theuser may select “Share With Metadata” button 408.

FIG. 5 illustrates an embodiment of a receiver interface 500 forreceiving a screen share 501 comprising captured metadata. Receiverinterface 500 comprises a chat window; however, receiver interface 500is shown for illustrative purposes only, a screen share 501 comprisingmetadata may be received in type of communication. Screen share 501corresponds to MyCapture 401 as is shown in FIG. 4, and comprisesselectable URL link 502, copyable search string 503, clickable link 502,and information regarding applications 505 and 506.

Referring again to FIG. 1, the following code segment shows anembodiment of computer code that may be embodied in metadata capturemodule 104 to capture all metadata in a window:

Windows using the ManagedSpyLib (available for free from MSDN): usingSystem; using System.Text; using System.Diagnostics; usingSystem.Windows.Forms; using System.Collections.Generic; usingMicrosoft.ManagedSpy; class Program {  static void Main(string[ ] args) {   Dictionary<int, int> topWindowCounts = new Dictionary<int, int>( );  foreach (ControlProxy proxy in ControlProxy.TopLevelWindows)   {    if(!topWindowCounts.ContainsKey(proxy.OwningProcess.Id))     {     topWindowCounts.Add(proxy.OwningProcess.Id, 0);     }    topWindowCounts[proxy.OwningProcess.Id]++;    }   foreach (int pidin top WindowCounts.Keys)   {    Process p =Process.GetProcessById(pid);   Console.WriteLine(“Process:”+p.ProcessName+“has”+    topWindowCount[pid].TString( ) + “top level windows”);   }  } }

The following code segment shows an embodiment an embodiment of computercode that may be embodied in metadata capture module 104 to capturemetadata for a button in a selected region:

foreach (Process p in Process.GetProcessesByName(“WindowsApplication1”)){   if (p.Id != Process.GetCurrentProcess( ).Id)   {    ControlProxyproxy =    ControlProxy.FromHandle(p.MainWindowHandle);   string val =(string)proxy.GetValue(“MyStringValue”);   .....   } }

The above code segments are shown for illustrative purposes only;metadata capture module 104 may implement any appropriate metadatarecognition technology.

FIG. 6 illustrates an example of a computer 600 which may be utilized byexemplary embodiments of systems and methods for metadata capture forscreen sharing as embodied in software. Various operations discussedabove may utilize the capabilities of the computer 600. One or more ofthe capabilities of the computer 600 may be incorporated in any element,module, application, and/or component discussed herein.

The computer 600 includes, but is not limited to, PCs, workstations,laptops, PDAs, palm devices, servers, storages, and the like. Generally,in terms of hardware architecture, the computer 600 may include one ormore processors 610, memory 620, and one or more input and/or output(I/O) devices 670 that are communicatively coupled via a local interface(not shown). The local interface can be, for example but not limited to,one or more buses or other wired or wireless connections, as is known inthe art. The local interface may have additional elements, such ascontrollers, buffers (caches), drivers, repeaters, and receivers, toenable communications. Further, the local interface may include address,control, and/or data connections to enable appropriate communicationsamong the aforementioned components.

The processor 610 is a hardware device for executing software that canbe stored in the memory 620. The processor 610 can be virtually anycustom made or commercially available processor, a central processingunit (CPU), a digital signal processor (DSP), or an auxiliary processoramong several processors associated with the computer 600, and theprocessor 610 may be a semiconductor based microprocessor (in the formof a microchip) or a macroprocessor.

The memory 620 can include any one or combination of volatile memoryelements (e.g., random access memory (RAM), such as dynamic randomaccess memory (DRAM), static random access memory (SRAM), etc.) andnonvolatile memory elements (e.g., ROM, erasable programmable read onlymemory (EPROM), electronically erasable programmable read only memory(EEPROM), programmable read only memory (PROM), tape, compact disc readonly memory (CD-ROM), disk, diskette, cartridge, cassette or the like,etc.). Moreover, the memory 620 may incorporate electronic, magnetic,optical, and/or other types of storage media. Note that the memory 620can have a distributed architecture, where various components aresituated remote from one another, but can be accessed by the processor610.

The software in the memory 620 may include one or more separateprograms, each of which comprises an ordered listing of executableinstructions for implementing logical functions. The software in thememory 620 includes a suitable operating system (O/S) 650, compiler 640,source code 630, and one or more applications 660 in accordance withexemplary embodiments. As illustrated, the application 660 comprisesnumerous functional components for implementing the features andoperations of the exemplary embodiments. The application 660 of thecomputer 600 may represent various applications, computational units,logic, functional units, processes, operations, virtual entities, and/ormodules in accordance with exemplary embodiments, but the application660 is not meant to be a limitation.

The operating system 650 controls the execution of other computerprograms, and provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices. It is contemplated by the inventors that the application 660for implementing exemplary embodiments may be applicable on allcommercially available operating systems.

Application 660 may be a source program, executable program (objectcode), script, or any other entity comprising a set of instructions tobe performed. When a source program, then the program is usuallytranslated via a compiler (such as the compiler 640), assembler,interpreter, or the like, which may or may not be included within thememory 620, so as to operate properly in connection with the O/S 650.Furthermore, the application 660 can be written as an object orientedprogramming language, which has classes of data and methods, or aprocedure programming language, which has routines, subroutines, and/orfunctions, for example but not limited to, C, C++, C#, Pascal, BASIC,API calls, HTML, XHTML, XML, ASP scripts, FORTRAN, COBOL, Perl, Java,ADA, .NET, and the like.

The I/O devices 670 may include input devices such as, for example butnot limited to, a mouse, keyboard, scanner, microphone, camera, etc.Furthermore, the I/O devices 670 may also include output devices, forexample but not limited to a printer, display, etc. Finally, the I/Odevices 670 may further include devices that communicate both inputs andoutputs, for instance but not limited to, a NIC or modulator/demodulator(for accessing remote devices, other files, devices, systems, or anetwork), a radio frequency (RF) or other transceiver, a telephonicinterface, a bridge, a router, etc. The I/O devices 670 also includecomponents for communicating over various networks, such as the Internetor intranet.

If the computer 600 is a PC, workstation, intelligent device or thelike, the software in the memory 620 may further include a basic inputoutput system (BIOS) (omitted for simplicity). The BIOS is a set ofessential software routines that initialize and test hardware atstartup, start the O/S 650, and support the transfer of data among thehardware devices. The BIOS is stored in some type of read-only-memory,such as ROM, PROM, EPROM, EEPROM or the like, so that the BIOS can beexecuted when the computer 600 is activated.

When the computer 600 is in operation, the processor 610 is configuredto execute software stored within the memory 620, to communicate data toand from the memory 620, and to generally control operations of thecomputer 600 pursuant to the software. The application 660 and the O/S650 are read, in whole or in part, by the processor 610, perhapsbuffered within the processor 610, and then executed.

When the application 660 is implemented in software it should be notedthat the application 660 can be stored on virtually any computerreadable medium for use by or in connection with any computer relatedsystem or method. In the context of this document, a computer readablemedium may be an electronic, magnetic, optical, or other physical deviceor means that can contain or store a computer program for use by or inconnection with a computer related system or method.

The application 660 can be embodied in any computer-readable medium foruse by or in connection with an instruction execution system, apparatus,or device, such as a computer-based system, processor-containing system,or other system that can fetch the instructions from the instructionexecution system, apparatus, or device and execute the instructions. Inthe context of this document, a “computer-readable medium” can be anymeans that can store, communicate, propagate, or transport the programfor use by or in connection with the instruction execution system,apparatus, or device. The computer readable medium can be, for examplebut not limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, device, or propagationmedium.

More specific examples (a nonexhaustive list) of the computer-readablemedium may include the following: an electrical connection (electronic)having one or more wires, a portable computer diskette (magnetic oroptical), a random access memory (RAM) (electronic), a read-only memory(ROM) (electronic), an erasable programmable read-only memory (EPROM,EEPROM, or Flash memory) (electronic), an optical fiber (optical), and aportable compact disc memory (CDROM, CD R/W) (optical). Note that thecomputer-readable medium could even be paper or another suitable medium,upon which the program is printed or punched, as the program can beelectronically captured, via for instance optical scanning of the paperor other medium, then compiled, interpreted or otherwise processed in asuitable manner if necessary, and then stored in a computer memory.

In exemplary embodiments, where the application 660 is implemented inhardware, the application 660 can be implemented with any one or acombination of the following technologies, which are well known in theart: a discrete logic circuit(s) having logic gates for implementinglogic functions upon data signals, an application specific integratedcircuit (ASIC) having appropriate combinational logic gates, aprogrammable gate array(s) (PGA), a field programmable gate array(FPGA), etc.

The technical effects and benefits of exemplary embodiments includeautomatic sending of metadata information in a screen sharingapplication without interrupting real-time communication.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A method for metadata capture for screen sharing of a graphical userinterface (GUI) screen, the method comprising: determining by a screenshare sending module a region of the GUI screen to be shared; analyzingthe GUI screen by a metadata capture module to determine metadatarelevant to the region to be shared; and sending a screen share of theregion comprising the relevant metadata to a receiver by the screenshare sending module.
 2. The method of claim 1, wherein the metadatarelevant to the region is located outside of the region.
 3. The methodof claim 2, wherein the metadata relevant to the region is locatedoutside of a visible region of the GUI screen.
 4. The method of claim 1,further comprising receiving the screen share of the region comprisingthe relevant metadata by a screen share receiving module, wherein thereceived screen share comprises the relevant metadata in the same formatin which it is presented in the GUI screen.
 5. The method of claim 1,further comprising displaying a list of the relevant metadata to asender of the screen share, receiving a selection of a piece of therelevant metadata from the sender, and sending the screen share of theregion with selected metadata.
 6. The method of claim 1, furthercomprising applying a set of stored preferences to the relevantmetadata, and sending the screen share of the region with metadatadesignated by the set of stored preferences.
 7. The method of claim 1,wherein the relevant metadata comprises authorship of a document,confidentiality information, audit trail information, table cell color,a persistent session ID, text content, text font size, text linespacing, window title, whether a window is minimized or maximized, or asearch string.
 8. The method of claim 1, wherein the screen share issent as part of a web conference or chat session.
 9. A system formetadata capture for screen sharing of a graphical user interface (GUI)screen, the system comprising: a screen share sending module configuredto determine a region of the GUI screen to be shared; a metadata capturemodule configured to analyze the GUI screen to determine metadatarelevant to the region to be shared; and wherein the screen sharesending module is further configured to send a screen share of theregion comprising the relevant metadata to a receiver.
 10. The system ofclaim 9, wherein the metadata relevant to the region is located outsideof the region.
 11. The system of claim 10, wherein the metadata relevantto the region is located outside of a visible region of the GUI screen.12. The system of claim 9, further comprising a screen share receivingmodule configured to receive the screen share of the region comprisingthe relevant metadata, wherein the received screen share comprises therelevant metadata in the same format in which it is presented in the GUIscreen.
 13. The system of claim 9, wherein the screen share sendingmodule is further configured to display a list of the relevant metadatato a sender of the screen share, receive a selection of a piece of therelevant metadata from the sender, and send the screen share of theregion with selected metadata.
 14. The system of claim 9, wherein thescreen share sending module is further configured to apply a set ofstored preferences to the relevant metadata, and send the screen shareof the region with metadata designated by the set of stored preferences.15. The system of claim 9, wherein the relevant metadata comprisesauthorship of a document, confidentiality information, audit trailinformation, table cell color, a persistent session ID, text content,text font size, text line spacing, window title, whether a window isminimized or maximized, or a search string.
 16. The system of claim 8,wherein the screen share is sent as part of a web conference or chatsession.
 17. A computer program product comprising a computer readablestorage medium containing computer code that, when executed by acomputer, implements a method for metadata capture for screen sharing ofa graphical user interface (GUI) screen, wherein the method comprises:determining by a screen share sending module a region of the GUI screento be shared; analyzing the GUI screen by a metadata capture module todetermine metadata relevant to the region to be shared; and sending ascreen share of the region comprising the relevant metadata to areceiver by the screen share sending module.
 18. The computer programproduct according to claim 15, wherein the metadata relevant to theregion is located outside of the region.
 19. The computer programproduct according to claim 15, further comprising receiving the screenshare of the region comprising the relevant metadata by a screen sharereceiving module, wherein the received screen share comprises therelevant metadata in the same format in which it is presented in the GUIscreen.
 20. The computer program product according to claim 15, furthercomprising displaying a list of the relevant metadata to a sender of thescreen share, receiving a selection of a piece of the relevant metadatafrom the sender, and sending the screen share of the region withselected metadata.